Multilingual Sentence Categorization According to Language 1 Categorization According to Language 1.1 from Text Categorization

Issues in sentence categorization according to language is fundamental for NLP, especially in document processing. In fact, with the growing amount of multilingual text corpus data becoming available , sentence categorization, leading to multilingual text structure, opens a wide range of applications in multilingual text analysis such as information retrieval or preprocessing of multilingual syntactic parser. The major diiculties in sentence cat-egorization are convergence and tex-tual errors. Convergence since dealing with short entries involve discarding languages from few clues. Textual errors since documents coming from diierent electronic ways may contain spelling and grammatical errors as well as character recognition errors generated by OCR. We describe here an approach to sentence categorization which has the originality to be based on natural properties of languages with no training set dependency. The implementation is fast, small, robust and textual errors tolerant. Tested for french, english, spanish and german discrimination, the system gives very interesting results, achieving in one test 99.4% correct assignments on real sentences. The resolution power is based on grammatical words (not the most common words) and alphabet. Having the grammatical words and the alphabet of each This Paper is published in the Proceedings of the European Chapter of the Association for Computational Linguistics SIGDAT Workshop \From text to tags : Issues in Multilingual Language Analysis" held March 95 in Dublin. language at its disposal, the system computes for each of them its likelihood to be selected. The name of the language having the optimum likelihood will tag the sentence | but non resolved ambiguities will be maintained. We will discuss the reasons which lead us to use these linguistic facts and present several directions to improve the system's classiica-tion performance. Categorization sentences with linguistic properties shows that diicult problems have sometimes simple solutions. Emergence of text categorization according to language came with the need of processing texts coming from all over the world. The goal of text cat-egorization is to tag texts with the name of the language in which they are written. Information retrieval is the main application eld. To do this job, the traditionnal way is to exploit the diierence between letter combinations in different languages (Cavnar and Trenkle, 1994). For each language, the system computes from a training set a proole based on frequency (or probability) of letter sequences. Then, for a given text, it computes a proole and select the language which has the closer proole. …